Support Transcriber session with Google realtime API #1321
Conversation
🦋 Changeset detectedLatest commit: dc5c258 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
jayeshp19
force-pushed
the
gemini-realtime-stt
branch
from
6407a82 to
663f44f
Compare
jayeshp19
force-pushed
the
gemini-realtime-stt
branch
from
663f44f to
aee4c1c
Compare
| @self._session.on("agent_speech_completed") | ||
| def _agent_speech_completed(): | ||
| self._update_state("listening") | ||
| if self._playing_handle is not None and not self._playing_handle.done(): |
There was a problem hiding this comment.
can you include comments on why this is needed?
| def _agent_speech_completed(): | ||
| self._update_state("listening") | ||
| if self._playing_handle is not None and not self._playing_handle.done(): | ||
| self._playing_handle.interrupt() |
There was a problem hiding this comment.
Why we should interrupt here?
There was a problem hiding this comment.
Because we call this function when speech is interrupted as well. They likely made some changes, but now Gemini returns server.turn_complete instead of server.interrupted when interrupted. It's confusing. In both cases, we are calling this function.
| from typing import Any, Dict, List, Literal, Sequence, Union | ||
|
|
||
| from livekit.agents import llm | ||
|
|
||
| from google.genai import types # type: ignore |
There was a problem hiding this comment.
The code here is hard to follow, really sad we don't have types (it's unclear what is the structure of the dicts)
| self._transcriber.on("input_speech_done", self._on_input_speech_done) | ||
| self._agent_transcriber.on("input_speech_done", self._on_agent_speech_done) | ||
| # init dummy task | ||
| self._init_sync_task = asyncio.create_task(asyncio.sleep(0)) |
There was a problem hiding this comment.
This isn't really doing anything?
|
Where do we make sure the transcribed user speech is inside the chat_ctx and always before the generated agent speech? |
Yes, it is being called directly from the base class. We need to keep the dummy task unless we wrap it with capabilities.support_truncate in the base class.
I don't think we need it, as the transcriber and LLM are independent of each other. |
I'm not sure to follow, the baseclass is utils.EventEmitter[EventTypes]
How do we get the user messages inside the ChatContext? |
I mean
from here when audio transcription is done- https://github.com/livekit/agents/pull/1321/files#diff-4b3e6842c9b1bf3130541b6b2fd18dcc7d1b0051285496eca0355e62938d13fbR351 |
What I mean here is that on some bad timings or if the VAD events are different, the data inside the chat context will not be "stable". E.g;
|
User audio is usually processed in real-time, and we receive transcriptions quickly. However, you're right that these scenarios can occur. |
Maybe we can just ignore for now and don't add them to the ChatContext. (Even if not ideal :/) |
jayeshp19
changed the title
jayeshp19
changed the title
No description provided.